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Foreword 


The  U.S.  Navy  routinely  conducts  airborne  expendable  bathythermograph 
(AXBT)  surveys  of  oceanic  regions.  Hundreds  of  AXBTs  are  expended  dur¬ 
ing  these  regional  surveys  and,  naturally,  some  of  the  profiles  contain  con¬ 
siderable  noise.  Because  it  was  costly  to  obtain  this  data,  it  is  unwise  to  discard 
profiles  because  of  noise  contamination.  Therefore,  it  is  desirable  to  develop 
a  cost-effective  method  for  processing  noisy  data.  This  report  describes  a  tech¬ 
nique  for  processing  noisy  AXBT  data  that  is  being  used  at  the  Naval  Ocean 
Research  and  Development  Activity. 


R.  P.  Qnorati,  Captain,  USN 
Commanding  Officer,  NORDA 


Executive  summary 


Noise  or  spikes  beyond  acceptable  levels  often  occur  in  data,  even  In  mi 
the  most  carefully  planned  and  executed  experiments.  A  method  is  desi  ribrd 
here  for  processing  noisy  temperature  profiles  obtained  from  airborne 
pendable  bathythermograph  surveys  to  a  level  in  which  useful  informal  mu 
can  be  gained.  The  technique  is  simple  to  apply  and  can  be  adapted  iu  a 
wide  range  of  noisy  data. 


1 

p 

t 


a 


I 

| 

8 


'!•! 

i 

3 


■5 

•1 

| 

1 


Acknowledgments 


Discussions  with  Z.  Hallock  of  NORDA  were  instrumental  in  the  develop¬ 
ment  of  this  processing  technique.  This  procedure  was  applied  and  critiqued 
In  J  Dastugue  of  Planning  Systems  Incorporated.  This  work  was  supported 
In  the  Office  of  Naval  Research  under  Program  Element  61153N,  Dr.  H. 
C  Eppert,  Jr..  Program  Manager,  as  part  of  Project  GEOSAT. 


) 


Contents 


A  technique  for  processing  noisy  AXBT  data 


6 


Introduction 

Fxtensive  airborne  expendable  bathythermograph 
i  AXBT)  surveys  are  routinely  conducted  by  the  (J.S  Navy. 
These  survevs  are  often  use-el  to  describe  mesoscale 
variability.  vveMern  boundary  eurrents,  or  e-delies.  Quasi 
sMioptu  coverage  ot  a-  oeeame  region  is  possible  with 
a  smell  1’  t  aircr  ,»ping  AXBTs  eiver  a  period  of 
three  to  1 1\ e  lull  flight  days.  Temperature  structure  is  at 
tamable  elown  10  lOO  m  by  using  shallow  probe-s  and  down 
tu  HtMi  m  by  using  deep  probes. 

Regional  AXBT  surveys  otte-n  e-xpe-nel  huneireels  ot 
'Will's,  and  some  of  the  profiles  comniemly  contain  con- 
-.iderahie  noise.  Noise  can  originate-  from  equipment 
malfunctions.  electrical  problems,  radio  interference',  rough 
sea  state,  or  weak  radio  signals  (due  to  transmission  range). 
Noisy  preihles  are  often  discarded  or  truncated  t H-ea use- 
editing  seems  impossible  or  very  costly.  Much  data,  which 
-a e-re  taken  at  considerable  expense,  are  not  processed. 

A  procedure-  tor  processing  such  AXBT  data  is  ete 
seritn-el.  The  teehnii)ue  is  simple  to  apply.  Very  little  user 
input  control  is  re-c)uired.  Data  editing  is  automatic  and 
produces  consistent  editing,  which  is  well  defined.  The 
leifmujiie  is  flexible  and  can  be  adapted  to  most  AXBT 
data  i whether  or  not  the-  data  are  noisy). 

Method 

Basically  i he  main  elements  ot  the  procedure  described 
hi  re  consisted  of  automatically  eeliting  the  cl.it a .  filtering 
iT"  data  in  ivmi  resolutions  (high  anti  low  j,  and  then  ceint 
1'ininu  the  appropriate  portions  of  the  low  resolution  profile 
aheie  law  data  were  very  noisy  with  the  high  resolution 
profile  where  raw  data  qualm  was  good  Stihsampling  ot 
>  uried  at  equal  intervals  m  the  resulting  profile.  This 
on  t lii cl  was  developed  at  the  Naval  Ocean  Rescan  h  and 
I  )i  veil  ipiiteiH  Activity  (NORI)A;  on  a  VAX  1 1,750  digital 
i  omputi-r.  and  was  first  applied  to  a  data  set  consisting 
"t  about  At H I  AXBT  profiles  collected  during  two  aircraft 
operations  All  routines  used  in  the  processing  scheme 
were  developed  earlier  as  general  utility  programs  lor  ap 
plication  on  a  standard  file  format,  whit  It  incorporates  all 
lu  phvsual  oceanographic  profile  data  holdings  ot  the 


Physical  Oceanography  Branch  at  NORDA  Thus,  tin 
runsti  earns  were  both  modular  and  flexible.  After  traiisfei 
ring  the  raw  data  from  digital  cassette  to  VAX  disk  Ides 
these  steps  were  followed  for  processing  the  rave  AXh  I 
data. 

1.  Plot  raw  temperature  data  versus  depth 

2. "  Auto  edit  the  raw  data;  insert  had  data  flags 

3. ’  Plot  edited  raw  data. 

i. "  Interpolate  bad  data  records. 

5.  Filter  raw  (edited)  data  to  desired  resolution 

(i. ’  Filter  raw  (edited)  data  to  a  degraded  down 
resolution. 

7.  Interpolate  higher  resolution  data  (Step  V  t,  ,,| 
propriate  levels. 

R. ’  Interpolate  lower  resolution  data  (Step  (>•  to  suit 
levels  as  Step  7. 

9.  ‘  Merge  appropriate  sections  of  the  interpolated  high 
resolution  and  the  interpolated  low  resolution  till 
for  the  final  profile. 

10  Plot  the  final  profile. 

Note:  ‘  indicates  optional 

Hath  step  of  the  processing  is  now  explained  in  detail 
The  example  AXBT  raw,  unedited  profile  shown  in  i  igun 
1  is  used  for  illustrative  purposes.  Data  h>r  tins  piofilt 
were  collected  at  H  1 1/„  which  corresponded  to  aUun  11 in 

sampling. 

Raw  ,  unedited  temperature  data  are  plotted  veisus  dc ■pi!i 
in  Step  1.  This  step  is  used  as  an  editing  guide  and  ,t‘< 
a  means  ot  intercomparison  with  a  final  plot  ot  pi... ;  , I 

data  (Step  10). 

Auto-editing  in  Step  2  consisted  of  removing  imp  > 
hie  data  values  and  then  straight  line  lining  scenic  no  i 
the  data  using  a  least  squares  method.  Data  that  dill,  1 
by  more  than  a  specified  tolerance  with  the  tutu!  .I.u  ■ 
were  replaced  with  had  data  flags.  Data  shown  m  I  igu; 

1  were  fitted  in  10  in  segments,  beginning  at  a  di-pili  t 
1  So  m.  and  were  replaced  with  had  data  flags  it  iliey  ,lu 
fered  bv  more  than  0.5"C  from  the  straigfit  line  lit  I  In 
resulting  profile  is  shown  in  Figure  2  (Step  V  Flu  litnm 
was  Ix-gun  at  1)0  m  to  avoid  removing  good  data  m 
regions  containing  strong  gradients  The  10  m  sei-nu  ,,i 
and  O.V’C  tolerances  were  determined  after  v.im  e\pe  i 
mentation  Segment  length  and  temper.uuiv  i  -liiiii. 


!i unfit  tuvil  adjustment  tor  application  in  regions  containing 
strong  gradients.  tor  particularly  noisy  profiles,  different 
segment  lengths  and  tolerances  could  be  required  within 
the  same  profile.  Step  2  would  be  skipped  if  inspection 
ot  the  profile  shown  in  Figure  1  fStep  1)  indicated  that 
no  editing  was  necessary  Then  only  Steps  5.  7,  and  10 
would  be  required  This  editing  technique  also  worked 
well  m  removing  isolated  data  spikes  and  was  found  to 
!v  suitable  tor  general  editing  of  AXBT  data. 

For  quaint  control,  it  was  important  that  the  edited 
si.tt  ,i  were  plotted  in  Step  3  before  missing  data  records 
wi  re  interpolated.  Gaps  evident  tn  the  protile  shown  in 
figure  2  resulted  from  consecutive  bad  data  records.  If 
flu  gaps  were  too  large  or  too  numerous,  the  profile  would 
have  been  cut  short  prior  to  the  gaps.  In  Step  4.  the  gaps 
m  the  edited  profile  were  interpolated.  This  step  was  re¬ 
quired  prior  to  tillering  the  data. 

Data  in  the  lower  noise  regions  (0-300  m  for  the  pro¬ 
file  in  Fig.  2 r  were  considered  good  to  at  least  4-m  resolu- 
' ion.  In  Step  3.  the  profile  was  low  pass  filtered  to  4-m 
resolution  using  a  filter  with  a  half-width  of  21  points, 
■i  Bartlett  taper,  and  a  cutoff  frequency  of  0.23  c/m  (Fig. 
'  l  >ata  in  the  higher  noise  regions  were  considered  good 
to.  at  lust.  20  tO  m  resolution.  Fortunately,  the  higher 
noise  areas  usually  occurred  only  below  the  thcrmodine 
legions;  vertical  structure  here  could  be  adequately  de- 
m  1 1 bed  with  lower  resolution  data.  Therefore,  the  profile 
was  tillered  to  a  'Him  resolution  in  Step  6  by  using  a  filter 
with  a  halt  width  ot  201  points,  a  Bartlett  taper,  and  a 
‘  utott  frequent  y  of  0.023  e/m  (Fig.  4).  This  filter  effec¬ 
ts  eiv  i  nns,ructed  the  temperature  profile  through  the 
i  enter  ut  the  high  noise  region. 

I»  was  advantageous  to  have  the  tinal  processed  data 
■  :  miegrai  multiples  of  I  m  m  depth.  For  the  filtered  pro 
ni'-s.  2  in  depth  intervals  corresponded  to  the  Nyquist 
iinplmg  interval  tor  the  I  m  resolution  data.  In  Steps 

uni  N  i  lie  t  m  resolution  protile  and  the  iO-nt  resolu 
•ion  profile  were  subsampled  at  2  m  levels.  In  Step  9  the 
m  i  '"•»  m  ot  the  i  ni  resolution  profile  were  merged  with 
’ G  i.Ca  trom  the  to  m  resolution  profile  below  3(H)  m. 

‘  >m  i  sampling  ol  the  protile  occurred  below  3()0  m.  The 
o' •ilium;  profile  is  show  n  m  Figure  3  (Step  10). 

Discussion 

Goals  ot  the  suoutitie  projei  t  would  not  have  been  ac 
•  mphshed  it  large  numbers  ot  profiles  had  been  rejected 
a  mini  and  Without  a  technique  similar  to  the  one 
a  s,  tihed  here .  approximately  10-13%  ut  the  profiles 
'■■mill  have  been  rc|ei  ted  llie  data  were  tar  too  noisy 


lor  techniques  based  on  manual  editing  or  on  tics:  . v 
terences  to  work. 

Manual  editing  ol  the  pruhles  would  have  Uen  n> . 
impossible  due  to  the  time  and  labor  required  tor  suHi 
editing.  In  many  cases  good  data  were  not  visu.div 
distinguishable  from  bad  data.  In  fact,  the  true  temporal  m  i 
seemed  to  he  the  mean  of  the  noise.  This  charai  n  r , si n 
is  probably  essential  it  the  method  is  to  work;  otherwise, 
the  data  could  he  biased  bv  the  bias  in  the  noise 

The  severity  ot  the  noise  problem  was  further  illusion  . ! 
by  the  spectrum  (Fig.  (>)  of  the  auto  edited  profile  web 
missing  data  interpolated  and  the  spectrum  (Fig.  7m}  il> 
final  combined  profile,  both  a:  raw  resolution.  The  -.pi, 
trum  for  the  edited  profile  was  unacceptable.  I  len  .  sp<  <  m  l 
energy  was  confined  to  one  energy  decade  tor  all  livqui  n 
des.  For  the  final  profile  the  spectrum  was  typical  lain  g 
was  down  by  more  than  five  decades  at  1  II/. 

Subsets  of  this  technique  were  used  on  all  the  profile- 
Manual  labor  was  approximately  the  same  for  processing 
clean  raw  data  as  for  very  noisy  raw  data.  A  little  mor< 
computer  time  was  needed  for  the  messy  data.  Manu.il 
edit ing  of  data  spikes  was  not  required  for  any  prolife 
Processing  progressed  quickly  and  smoothly,  and  was  i ve¬ 
to  document.  User  input  control  was  minimal 

Filter  parameters  (Steps  3  and  6)  and  edit  paranien  e 
(Step  2)  were  determined  tor  the  entire  data  set  alter  ali 
raw  data  had  been  plotted.  When  the  processing  of  a  pi- 
file  required  all  10  steps,  as  determined  from  the  raw  pn  -lib. 
plot  in  Step  1.  the  only  parameters  to  he  determined  wen 
the  depth  range  over  which  to  apply  the  auto  editor  Si.  , 
2)  and  the  depth  at  which  to  recombine  the  higher  ,u-d 
lower  resolution  profiles  (Step  9).  If  the  profiles  reqtui  i  c 
spike  editing  (Step  2)  but  did  not  require  degrading  th, 
resolution  (Step  (0.  then  Steps  6.  S,  and  9  could  Is  \kipp< 
Then,  only  where  to  apply  the  editor  needed  to  be  di  u  < 
mined.  It  the  profiles  did  not  require  editing,  then  No  • 

2.  3.  i.  (>.  H,  anil  9  could  he  skipped.  Parameters  ri  < i>n: 
lor  processing  clean  profiles  were  the  same 

Summary  and  conclusion 

Complete  processing  consisted  ot  three  phase--  I  b 
the  AXBT  data  were  transferred  from  digital  i ass.  :t. 
the  VAX  1 1/730  computer.  depth  and  temperature  a .  i, 
computed,  and  the  d  tta  were  written  in  the  standard 
format.  Then,  raw  data  were  plotted  for  all  profiles  I  n-  m 
based  on  the  information  gained  from  the  raw  pb  i 
(for  noisy  raw  data)  or  part  (for  ilean  raw  data  ot  S'.  . 

2  10  were  executed  in  a  single  computer  job  TIiim 
included  auto-editing.  filtering,  interpolating,  and  pi.  n: 


of  the  data.  A  different  filtering  procedure  was  applied 
to  the  noisier  portions  of  the  profiles.  If  this  filtering  pro¬ 
cedure  was  applied,  final  processed  profiles  were  at  a  lower 
resolution  over  the  depth  range  where  noise  was  a  problem 
than  user  the  depth  range  where  noise  was  not  a  problem. 
Often,  noisy  areas  were  limited  to  portions  of  the  profile 
that  could  be  adequately  described  at  lower  resolution. 

This  technique  salvaged  many  questionable  profiles.  The 
processing  was  quite  flexible,  simple,  and  easy  to  use.  In 
general,  it  can  be  used  in  processing  a  wide  variety  of 


AXBT  data.  Similar  techniques  can  be  applied  to  mlai 
data  types. 

Six  general,  nondata-specific  utility  programs  were  used 
in  Steps  2-10.  Conceivably,  these  steps  could  be  combined 
into  a  single  program  (or  two.  if  plotting  programs  were 
separate).  However,  such  a  program  could  be  quite  huge, 
complicated,  and  data  specific.  Fully  automating  this  pro 
cedure  is  possible.  Decisions  concerning  where  to  apply 
the  editor  and  where  to  recombine  the  profiles  can  tv  made 
via  programming  logic.  More  work  is  required  in  this  ate.t 
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